Exploiting Image Descriptions For The Generation Of Referring Expressions

Intelligent multimedia representation systems (e.g. (Wahlster et al., 1993), (Andre et al., 1996), (Feiner and McKeown, 1993)) have to select appropriate expressions in different modes (texts, graphics and animations) and to coordinate them 1. For both tasks, an explicit representation of the content of the multimodal expressions is required. An important aspect of the media coordination is to ensure the cohesion of the resulting multimodal presentation. One way to tie the expressions in the different modes together is to generate referring expressions using co-referential relations between text and graphics. In order to construct appropriate referring expressions for the displayed objects in the graphics, one has to choose what attributes of the objects could be used for constructing an unambiguous linguistic realization. Most of the algorithms proposed by other researchers (e.g. (Dale and Reiter, 1995)) use information on the type of the object and perceptually recoguisable attributes like colour or shape. Some systems exploit additional information as descriptors such as the information on complex objects and their components (IDAS (Reiter et al., 1995)) or the spatial inclusion relation (KAMP (Appelt, 1985)). However, other kinds of descriptors, such as information on the relative location of a component with respect to another, have not been used yet. In this paper, we propose an algorithm to compute a set of components for sides of complex ob-